Parallelizing Over Artificial Neural Network Training Runs with Multigrid
نویسنده
چکیده
Artificial neural networks are a popular and effective machine learning technique. Great progress has been made on speeding up the expensive training phase of a neural networks, leading to highly specialized pieces of hardware, many based on GPU-type architectures, and more concurrent algorithms such as synthetic gradients. However, the training phase continues to be a bottleneck, where the training data must be processed serially over thousands of individual training runs. This work considers a multigrid reduction in time (MGRIT) algorithm that is able to parallelize over the thousands of training runs and converge to the same solution as traditional serial training. MGRIT was originally developed to provide parallelism for time evolution problems that serially step through a finite number of time-steps. This work recasts the training of a neural network similarly, treating neural network training as an evolution equation that evolves the network weights from one step to the next. Thus, this work concerns distributed computing approaches for neural networks, but is distinct from other approaches which seek to parallelize only over individual training runs. The work concludes with supporting numerical results that demonstrate feasibility for two model problems.
منابع مشابه
Optimizing of Iron Bioleaching from a Contaminated Kaolin Clay by the Use of Artificial Neural Network
In this research, the amount of Iron removal by bioleaching of a kaolin sample with high iron impurity with Aspergillus niger was optimized. In order to study the effect of initial pH, sucrose and spore concentration on iron, oxalic acid and citric acid concentration, more than twenty experiments were performed. The resulted data were utilized to train, validate and test the two layer artificia...
متن کاملLearning Curve Consideration in Makespan Computation Using Artificial Neural Network Approach
This paper presents an alternative method using artificial neural network (ANN) to develop a scheduling scheme which is used to determine the makespan or cycle time of a group of jobs going through a series of stages or workstations. The common conventional method uses mathematical programming techniques and presented in Gantt charts forms. The contribution of this paper is in three fold. First...
متن کاملArtificial neural network forecast application for fine particulate matter concentration using meteorological data
Most parts of the urban areas are faced with the problem of floating fine particulate matter. Therefore, it is crucial to estimate the amounts of fine particulate matter concentrations through the urban atmosphere. In this research, an artificial neural network technique was utilized to model the PM2.5 dispersion in Tehran City. Factors which are influencing the predicted value consi...
متن کاملEvaluation of Ultimate Torsional Strength of Reinforcement Concrete Beams Using Finite Element Analysis and Artificial Neural Network
Due to lack of theory of elasticity, estimation of ultimate torsional strength of reinforcement concrete beams is a difficult task. Therefore, the finite element methods could be applied for determination of strength of concrete beams. Furthermore, for complicated, highly nonlinear and ambiguous status, artificial neural networks are appropriate tools for prediction of behavior of such states. ...
متن کاملParallelizing Linear Recurrent Neural Nets over Sequence Length
Recurrent neural networks (RNNs) are widely used to model sequential data but their non-linear dependencies between sequence elements prevent parallelizing training over sequence length. We show the training of RNNs with only linear sequential dependencies can be parallelized over the sequence length using the parallel scan algorithm, leading to rapid training on long sequences even with small ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1708.02276 شماره
صفحات -
تاریخ انتشار 2017